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EVALUATION OF PERFORMANCE FOR UNIVERSITY RESEARCH SCIENTISTS. 



Research scientists go to great lengths to ensure that their work is 
objective. They design experiments to test hypotheses, put the variables or 
conditions into effect, and count, measure or collect samples to determine the 
outcome. The results are summarized, often subjected to statistical analysis and 
prepared for publication. A manuscript is usually internally reviewed and then 
submitted to a journal for external review. 

During this process there are guides to keep researchers on the right path. 
^Te^usually discuss our research with others while it is in the planning stage and 
get opinions about our ideas. The scientific method provides a framework for the 
conduct of the research, modified to meet the particular practices of our 



and is usually fairly comprehensive and objective. 

Evaluation of the performance by a research scientist should have many of 



performance which are being used. Eight characteristics of effective standards 
[1] have been summarized: 




the same characteristics. Each scientist should know the standards of 



1. They are based on the job and not the person(s) in the job. 

2. They are achievable. 

3. They are understood. 



4. They are agreed on. 



5. They are as specific and measurable as possible. 
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6. they are time-oriented. 

7. They are written. 

8. They are subject to change. 

The premise is that effective standards increase the performance of an 
individual. If the evaluator and scientist being evaluated jointly develop the 
standards, fewer problems occur. The evaluator and scientist will both 
understand the goals of the institutional unit and also the objectives of the 
scientist. Scientists can then devote all of their energy to accomplishing 
objectives rather than divert some energy to second-guessing what an evaluator 
really expects [2]. When openness and merit are cornerstones of evaluation, 
there should be no surprises and more positive emotions at the time of 
performance evaluation [3] . 

Standards of performance can be developed for scientists engaged in 
research activities at a university. Some scientists object to a detailed 
evaluation, lest ^/e^pecome "bean counters". However, research scientists are by 
training and aptitude bean counters in their profession, because they evaluate 
and solve problems by counting and measuring. During evaluation of a scientist, 
beans are always being counted in assessing performance and distributing rewards. 
In this process, it is important that the evaluator distinguish between a lima 
bean and a soybean and that the scientist being evaluated knows what -the criteria 
are . 

Research scientists have two missions. One is to add value by solving 
problems. Applied research is aimed at removing obstacles that have an immediate 
economic impact. Basic research is focused on solving important scientific 
questions that may not have immediate or foreseeable economic significance. 
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A second mission of research scientists is to train future scientists. 
Graduate students must be guided in acquiring knowledge that is already known in 
a discipline. They must be coached in learning skills and procedures that will 
enable them to acquire new knowledge and make it available to others. Adviser 
assistance is usually needed in procuring their first position. 

When estimating the value added by a researcher, it is important to 
distinguish between activities and results [4] . An analogy can be drawn from the 
exploration for oil. Efforts are made to maximize the possibility for locating 
a profitable source of oil. These include an examination of geologic 
characteristics, seismic exploration and an assessment of other wells in the 
area. When a promising site is located, a well is drilled and casing is put in 
place. All of these activities are an expense. A flow of oil large enough to 
pay expenses and yield a profit is the desired result. Too many unsuccessful 
attempts will bankrupt a company. 

Research also has activities and results. Planning research, writing grant 
proposals, conducting experiments and preparing manuscripts are examples of 
activity. Being awarded a research grant or solving an important problem are 
examples of results. The value of the results must outweigh the cost of the 
activities for long term success. 

Evaluations should be made by those with the qualifications and willingness 
to make judgements about the scientific contributions of an individual . Part of 
the evaluation can be based on quantity and part should be based on quality of 
production. Scientific discoveries from a laboratory are different from cars 
from an assembly line. Each car has approximately the same value, so it is 
important to produce as many similar cars as can be sold. In contrast, a 
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scientist's lifetime contributions can be fairly accurately gauged by examining 
the 10 best publications. 

Performance evaluation should be based on several criteria related to 
quantity and quality. For applying standards of performance, job segments can 
be identified [1] for research scientists. Such segments might be to: 

1. Investigate and solve problems pertinent to your area of expertise. 

2. Disseminate information gained from research through journals, books 
and magazines and by presentations at meetings. 

3. Develop extramural sources of funding for research. 

4. Recruit and develop graduate students. 

5. Develop a service component by serving on appropriate committees and 
editorial boards. 

Impact of Research Findings 

Assessing quality is much more difficult than assessing quantity. A model 
that might serve as a model to assess quality is described in a Taxonomy of 
Educational Objectives [5]. This model deals with recalling knowledge and 
developing intellectual capabilities. Six major categories of knowledge, 
comprehension, application, analysis, synthesis and evaluation are developed. 
The premise is that these categories exist in a hierarchy and that greater 
intellectual abilities are required to complete higher order tasks. Higher order 
tasks are more difficult to complete, but successful completion of the higher 
order tasks adds more value. 

Scientific contributions might also be considered in terms of hierarchical 
order. Some scientific discoveries add more value than others. One might 
visualize scientific discoveries as breaching a series of walls. A wall of 
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ignorance separates the known from the unknown. Only a few individuals have the 
knowledge, technical ability and intuition to visualize what is beyond the wall. 
However, once the wall is breached, a larger number of individuals can 
participate. Some will rush through to explore the new territory that is now 
accessible, while others may dismantle the wall in order to build something new. 
Eventually a new wall will be encountered and the process repeated. Details of 
the genesis of one important discovery have been shared [6]. 

To develop the concept of hierarchical value, an example might be drawn 
from the study of vitamins. In 1900, scientists were not aware of an animal's 
need for nutrients called vitamins [7]. The need for sources of carbohydrate, 
protein, fat and some minerals was known. During the following years it became 
apparent that foods contained nutritional factors that were needed in very low 
concentrations to prevent or cure disease. 

These initial discoveries breached a wall of ignorance about vitamins and 
opened the way for numerous other discoveries. Realizing that the discovery of 
the first vitamins suggested that there were more, scientists continued to search 
for and find vitamins over a period of approximately 50 years. Discovery of 
these later vitamins might be considered one order lower in the hierarchy than 
discovery of the original concept of vitamins. Accomplishments such as 
discovering the structure, the mode of action, and the nutritional requirement 
in the first species of animal might be assigned the' next lower order. 
Determining the requirement in the diet for other species of animals might be one 
order lower. 

The value of each order in relation to others does not appear to be an 
arithmetic progression. Value added by each successive order might more 
accurately be described by a logarithmic progression. Research that determines 
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the dietary nutrient requirements of an animal might be assigned a value of 10°. 
Discoveries in the ascending orders might be assigned values of 10 l , 10 2 , and 10 3 . 
Papers that detail discoveries at the top of the hierarchy add an immense amount 
of value to basic research. 

Paths to discoveries at the top of the hierarchy are varied. In some 
cases, researchers have stalked a scientific answer for decades, producing a long 
publication record. In other cases, the first-listed author had almost no 
publication record. The discoverers of insulin [8], structure of DNA [9], and 
polymerase chain reaction [10] fit into this category. 

For evaluating the impact of research findings , there are now quantitative 
aids. One way is to count how often an author's research has been cited during 
the past few years [11]. This method, which is a summary of short-term interests 
of scientists, would probably not have evaluated the importance of "jumping 
genes" [12] . The count could be based on citation rate by other scientists only, 
or on citation rate by other scientists plus citation by an author of his/her own 
research. The author of a well-accepted method or review article generally fares 
well using this method of evaluating impact. 

Another quantitative aid for evaluating impact of research is to assign an 
impact factor to journals [13]. If research is published in a journal with a 
high impact factor, the research may be regarded as of better quality than 
research published in a journal with a low impact factor. While there is 
validity to this approach, it is important to avoid the stereotype that all 
research published in journals with high impact factors has a high impact and 
that all research published in journals with a low impact factor will have a low 
impact . 
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Some questions that need to be answered about the impact of research 
findings are: Where does the research of the scientist appear to fit in the 
hierarchy of that particular area of expertise? If the problem is solved, how 
much value will be added to the economy or to the knowledge of basic science? 
What is the rate of progress on solving the problem? Sometimes a well-designed 
and promising plan of attack on a problem yields little useful information. Such 
an outcome would be evaluated considerably less favorably than an outcome that 
adds a large amount of value. An element of risk enters at this point. Should 
a scientist attempt research with low risk and low return or should research be 
attempted that has the possibility of high risk and high return? The results, 
and not the activities, should be evaluated. 

Number of Publications 

Authorship of scientific publications is usually the main standard to 
estimate quantity of research. If this is done, the relative value of different 
types of publications should be stated. For certification purposes, the relative 
value assigned to different types of publications [14] is as follows: 



Research/review paper in peer-reviewed journal 1 
Commercial books (>96 pages) 2 
Book chapters ..8 
Abstracts/posters at meetings or conferences " .4 
Case report .3 



The relative value of publications could be different for each administrative 
unit but should be uniformly applied to all researchers. The number of 
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publications of each type can be multiplied by relative value to determine 
publication units(PU), which can then be added. 

Concerns have been expressed about the importance that is sometimes 
attached to numbers of publications. Reviews of an individual for promotion and 
tenure, for a grant proposal, or for recognition are often greatly influenced by 
the quantity of the publication record, rather than by the quality. A desire to 
expand the publication list leads to questionable practices. One has been a 
significant increase in number of authors per journal article [15]. This may 
occur because present research problems are more complicated and require more 
areas of expertise and/or because authorship is unjustified. A second problem 
is wasteful publication. Results of a study may be subdivided into several 
publications rather than presented in one more extensive article, or the same 
study may be presented in different journals by different authors [16]. In 
addition, approximately half of published papers were not cited in the five years 
after publication [17]. Each time the peer-review and publishing process is 
engaged, costs are incurred in time and money. 

Some institutions have taken steps to decrease the importance of quantity 
and increase the importance of quality during the process of evaluation. A 
suggestion was to limit publications to the three best articles in a year, with 
no more than 10 for a five-year period [18]. The National Science Foundation 
permits only five publications listed for a grant application [17]. This 
limitation directs attention to the quality of research. 

For someone outside the researcher's administrative unit, opinions about 
a scientist's research may be based only on counting publications. Results of 
different ways of counting have been presented [19]. However, within an 
administrative unit, an annual evaluation of a research scientist should be more 
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detailed. The acceptance of a peer-reviewed paper is based upon its content, not 
upon the number of authors contributing. Evidence of this fact is that many 
journals have instituted a review process that includes removal of the names of 
authors and their affiliation. The number of authors has no influence on the 
acceptance or rejection of the manuscript. It is the contribution of the 
manuscript that is judged. Therefore, the manuscript becomes the PU, and the 
authors should be credited a fraction based on their contribution. The total 
number of PUs in an administrative unit should be the same if summed from types 
of publications or from contributions of individual scientists. 

Another benefit results from assigning scientists a fractional value of a 
publication. Most university research scientists have responsibilities for more 
than one type of scholarship [20] . Teaching has not experienced the inflation 
in number of teachers per course that .research has experienced in number of 
authors per paper. Therefore, using deflated numbers for both teaching and 
research permits a fair evaluation of a faculty member's contributions. 

It is possible to compare the capabilities required for various types of 
evaluation. Suppose that a list of 100 scientific publications, together with 
authors, was compiled from a department. The most simple type of evaluation 
would be to search for an author's name and count one additional number each time 
that author's name is located. Most children who have completed the second grade 
are capable of such evaluation. 

If someone was to assign a fractional value to each publication or author 
of a publication, additional skills would be needed to complete the evaluation. 
Children with mathematics competency of sixth or seventh grade would be needed 



to add the fractions. 
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If the evaluator should be able to read and comprehend the Information in 
the publications, additional capabilities are needed. Upperclassmen studying 
toward a Bachelor's degree could generally complete such an evaluation. 

If the evaluator is expected to understand information in a publication and 
make a judgment about the value added by the information, a higher level of 
capabilities is required. Some, but not all, scientists with a Ph.D. are capable 
of such evaluation at the time they receive the degree. It is appropriate that 
higher order thinking be applied for a significant portion of the evaluation. 
Scientists, like others, tend to live up to, or down to, expectations. 

Resource Self-sufficiency 

Research is expensive. Minimum costs include the salary and fringe 
benefits of the scientist, and they should be evaluated on the basis of quantity 
and quality of contribution. Almost certainly there will be additional personnel 
costs for secretarial and technical assistance. Properly equipping a laboratory 
for some types of research is a major expense. Ongoing research consumes 
supplies and services which may be very expensive. 

Money for research activities may come from intramural or extramural 
sources [21], Intramural funds, mostly from tax sources, are received by the 
institution and distributed to support research. They typically- support the 
barest essentials of a research program. Some research scientists are able to 
solve problems and add to knowledge with only a meager budget, while other 
scientists require a much larger budget. Those who require additional funds to 
conduct research usually must acquire them from extramural funds. Competitive 
grants from government agencies supply most of this money, but lesser amounts are 
received from business and industry and other private sources. 
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The ideal situation for an institution is to be affiliated with a 
productive scientist who costs the institution nothing financially. To determine 
the costs incurred by each scientist, a system of record keeping is required. 
Costs of personnel, equipment and supplies must be determined. The ability of 
a scientist to offset these costs with extramural funds can be determined. In 
this manner a scientist would be compared to his peers for efficiency of using 
research funds. 

The research scientist then has choices. Spending more time to attract 
extramural funds may detract from excelling in other job segments. Building a 
larger research program will cost more than building a smaller one. The research 
scientist must make choices about the combination of effort and money that will 
result in a most advantageous evaluation. For the evaluator and scientist, this 
part of the evaluation adds the dimension of accountability for the use of 
financial resources. 

Graduate Education 

Developing the next generation of scientists is also a mission of research 
scientists that should be evaluated on the basis of quantity and quality of 
contribution. The adviser is important in guiding a student through the 
transition from an undergraduate student to an advanced degree holder. Many 
undergraduates are passive consumers of information. As ' they develop, they 
become producers of information. The adviser needs to provide individual 
attention so that the student can succeed in his/her academic program, can 
contribute to projects within the academic unit, and can learn the process of 
solving problems. 
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The ability to recruit well-qualified students is an important attribute 
of a scientist. A student with a good academic background, good intellectual 
abilities and good people skills can quickly make contributions within the 
academic unit. Students with deficiencies in one or more of these areas take 
longer to develop to the point where they are net contributors to the system. 
More adviser effort is required to assist the progress. 

Another characteristic of a good adviser is the ability and willingness to 
develop the talent a student has. Students should not be accepted for graduate 
study if they do not appear capable of completing a degree. However, once 
accepted, the student should receive appropriate advice and encouragement that 
will enhance the probability of successfully completing a degree. Students may 
take slightly different routes and different lengths of time to complete the 
degree, but they should all meet appropriate .minimum standards. Although an 
adviser should be an advocate for the student, if it becomes evident that a 
student cannot meet the requirements of the degree, then the adviser should act 
to bring about an amiable conclusion to the partnership. 

The adviser can also be judged on the success of former students. If 
quality students enter a quality graduate program, receive quality advising, and 
enter a job market that has a scarcity of people, chances for career success are 
very high. If one or more of those conditions is less than optimal,- chances for 
career success decrease. The adviser can have an important influence during the 
period of graduate study and in the assistance to acquire a first position. Some 
of the success of former students should rightfully be attributed to the adviser. 
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Service 

A small but specified portion of the evaluation should be based on service 
activities. There are small jobs that need to be done in organizations that may 
benefit others more than the one who completes the job. If these jobs are evenly 
distributed, then no one is burdened. If some fail to do their fair share, 
others may accept more than their fair share in order to accomplish what is 
needed. 

The research scientist should be informed of the relative value attributed 
to various service activities. How does the importance of chairing the Graduate 
Studies Committee in the unit compare to serving as a reviewer for a journal? 
What about responsibility for organizing sessions at national meetings or 
specialized conferences? Should one serve as a section editor or editor of a 
journal? How important is it to become an officer of a national or international 
organization? 

Service activities that require more time and have more responsibility 
require more planning. If the scientist has the desire, does the evaluator 
encourage such a commitment? If so, years of development and participation are 
usually needed before achieving such an end. 

Just as in the other areas of evaluation, an individual's service 
performance may range from outstanding to poor. Standards for service should 
have dimensions for quality and quantity. 

Defining Expectations 

Differences of opinion exist, and are often strongly defended, about job 
segments to be evaluated, standards of performance to be used in evaluating each 
segment, and relative weight assigned to each segment. The basis for these 
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decisions should be the mission of the administrative unit. Some difference will 
probably exist among individuals in relative weight assigned to each segment; 
however, the fractional contribution by each individual should add up to the 
desired effect for the total unit. For evaluators who are more authoritarian, 
these decisions will be made by the evaluator. For evaluators who are more 
democratic, these decisions can be made after discussions with the whole group 
of scientists and then with individuals. Because the objective of evaluation is 
to improve job performance, the above information should be clear to all who are 
being evaluated. 

The evaluation suggested requires time and effort. For the evaluator, a 
guide might be as follows: calculate the average salary of those of similar 
academic rank in the unit; express your salary as a percentage of that average 
salary; subtract 100 to find the percentage of time to be devoted to evaluation. 
While this amount of time may appear excessive, it can be productive use of time. 
Spending an appropriate amount of time and effort collecting information for 
evaluation of scientists will probably result in a more productive unit that can 
solve problems more efficiently and train future scientists better. 

SUMMARY 



Guidelines for evaluating university research scientists are described. 
Each scientist should know the standards of performance that are used in 
evaluation, the job segments that are being evaluated, and the relative weight 
assigned to each job segment. Five job segments have been identified i 
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1. Impact of research findings 

Research efforts will produce varying amounts of impact. In most 

disciplines, a hierarchy of impact can be constructed. Findings that are 
closer to the top of the hierarchy have substantially more impact than 
findings at the bottom. Important discoveries in the basic sciences 
generally uncover new areas for research. Important discoveries in 
applied research usually solve problems that have a large economic impact. 
Evaluation of this component is based on prior discoveries and on the 
likelihood of significant discoveries from ongoing research. 

If rate of citation or impact factor of a journal is used to 
establish impact of research, details of the procedure should be specified. 

.2. Number of Publications 

A prescribed value is attached to each class of publication. If a 
referred journal article is assigned a publication unit of 1, then books 
could be assigned a value of 2, book chapters .6, abstracts/posters at 
meetings .4 and case reports .3. Each contributing author is then 
assigned a fraction of the publication unit based on proportion of 
contribution. 

3. Resource self-sufficiency 

Research is expensive and money should be used efficiently. To encourage 
efficiency, the money that an institution uses to support a research 
scientist can be compared to the money that a research scientist is able 
to obtain from extramural funds. The salaries of the scientist and 
supporting personnel are costs. The cost of supplies, equipment, and 
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services from the institution should be included. These costs could be 
compared to revenue from extramural funds and some product that might be 
generated as a result of the research program. The net result from 
individual scientists can be compared. 

4. Graduate education 

Training future scientists is an important function of research 
scientists. The ability to recruit well-qualified students, develop their 
abilities and assist them in acquiring appropriate positions should be 
evaluated. Numbers of students advised to graduate degrees and quality of 
the students are important considerations. 

5. Service 

Some activities and duties benefit organizations more than individuals. 
Research scientists should be willing to assume their fair share of 
activities such as Graduate Studies Committee Chairman, reviewers for 
journals, and service in national organizations. Quantity and quality of 
contributions should be considered. 
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